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ABSTRACT 


This thesis examines the relationship between weapon 
system acquisition program's initial conditions (project 
size, technical risk and program length) and program 
outcomes in the areas of cost, performance and schedule. 
The study employs a nonparametric correlation proceudre 
and Mann-Whitney U Tests as the principal analytic tools 
of the examination process. 

The results of the analysis indicate that a definite 
relationship between cost/schedule growth and program size 
exists. In a significant number of cases, large programs 
incurred greater absolute cost growth with less schedule 
slippage than did programs of smaller size. The variance 
in project technical performance is largely unexplained 
by the Weaedoilsas: although there are indications that 


it is related inversely to program length. 
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Lt.  ENDRODUCTION 


The recent draft of a revision to Department of Defense 
(DOD) Directive 5000.1 suggests that the Pentagon is moving 
into a new era of program management. The groundwork for 
this revision to David Packard"*s earlier attempt to establish 
a firm policy base for the DOD procurement has been well 
laid by the reports of the Commission on Government Procure- 
ment and the Acquisition’ Advisory Group, and by Office of 
Management and Budget Circular A-109. This modification of 
Directive 5000.1 places greater emphasis on the necessity 
to evaluate proposed new weapon systems on the basis of 
reconciled mission needs and requires an acquisition process 
in which alternative concepts/solutions are investigated in 
detail. The degree of program oversight by the Office of the 
Secretary of BecGtise (OSD) as proposed has been increased by 
the addition of another decision milestone, “DSARC O", at 
Which point the initiation of the conceptual effort will 
require Secretary of Defense approval. 

Faced with mounting public pressure for greater control 
of costs and performance following some notable examples of 
forecasting errors like the C-5A and LHA programs, DOD 
appears to have responded by striving to exorcise every 
possible bit of uncertainty from the conduct of weapon 
system acquisitions prior to the committment of major 
resources. The desired result would be the procurement of 


systems which satisfy basic operational requirements by a 





firm date and at a reasonable and predetermined price. 
Although written several years prior to the proposed 
policy update, J. Ronald Fox's words have an 'I've-seen-it- 
all-before' air to them as he comments in Arming America on 
the changing patterns of military acquisition management. 


Since 1969 there have been further attempts to 
improve other aspects of weapons acquisition 
management. These include the establishment of 
"milestones" for major programs (i.e., dates for 
the accomplishment of crucial program objectives) ; 
the synchronization of management personnel 

changes with program milestones; a reduction in 

the numbers of letters of contract; the improvement 
of training programs for program management 
officials; a formal review procedure for each new 
weapon system's operational specifications. The 
object of the review would be to insure that 
program requirements are reasonable and attainable, 
and that prototype systems are developed and 

tested before a production program is approved. 
Serious doubts about the successful implementation 
of this proposal have troubled Defense Department 
observers, however. . .Many procurement officials 
in the Defense Department are, after all, aware © 
that the present system has faults. But despite 

a steady succession of studies and recommendations, 
the procurement process has remained impervious to 
structural reform.1l 


Gexmtaindy only time will prove Fox right or wrong, but there 


1s some evidence submitted by OSD that cost control has at 


least received greater attention among the major programs. ? 


1 Fox, J. R., Arming America, p. 456, Harvard University 
Press, 1974 


2 In a memorandum Originating in the Office of the 
Director, Planning and Evaluation (OSD) dated 9 August 1976, 
it was reported that weapon system cost growth had declined 
generally in the last twelve months. 





The purpose of this study was twofold. The first was 
to establish an historical perspective of major programs 
reaching fruition in the early to mid-1970's which will 
provide a means of comparison for future studies of program 
success, and thus perhaps support Fox's view. While the 
meaning of “program success" might well vary, it is here 
defined as the degree to which a program met initial 
estimates of cost, schedule and performance criteria. The 
first goal was met essentially by listing those programs 
studied with their associated degrees of success in 
Appendix C. 

The second desired result was to investigate the validity 
of certain hypotheses about program management. Specific- 
ally, the approach taken in this case was to highlight gross 
differences in program outcomes on the basis of areca 
program characteristics, such as program size, initial 
funding and technical advance. Associated with this aim was 
an extensive correlation analysis in which the predictability 
of program success was determined from program initial 
conditions when certain variables such as service sponsor, 


weapon system type and life-cycle phase were controlled. 


A. UNCERTAINTY IN WEAPONS SYSTEM DEVELOPMENT 

A. W. Marshall and W. H. Meckling of the Rand Corporation 
addressed the problem of the predictability of cost, 
schedule and performance estimates associated with development 


programs in a report prepared for the U. S. Air Force in 1959. 





Their conclusions regarding the uncertainty of new develop- 
ments were: 
1. Early estimates of important parameters are usually 
quite inaccurate in two respects. First of all, they 
are biased toward overoptimism, and secondly, their 
error is subject to a large variation. : 
2. The accuracy of estimates is a function of the 
stage of the system life-cycle at which time the 
appraisal was made. 
The authors suggested a relationship between developmental 
estimates and the military world of R&D decision making, 
but little if any analysis was provided in their report 
other than to say that a problem in the way estimates are 
generated does exist. Also, the number of programs 
reviewed appeared rather small (22) and they were all Air 
Force sponsored. 

Following the above report was the work of M. J. Peck 
ana Ff. @. Scherer, The Weapons Acquisition Process: An 
Economic Analysis which was published in 1962. This bell- 
wether study of the business of arms procurement improved 
noticeably on Marshall and Meckling's efforts to gain an 
understanding of the unpredictable nature of weapons 
acquisition. 

A major thesis of this study is that the weapons 
acquisition process is characterized by a unique 
set of uncertainties which differentiates it from 


other economic activity. To be sure, uncertainty 
is a pervasive feature of all economic activity, 
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and most of the uncertainties in weapons acquisition 
have their commercial counterparts. But there is a 
uniqueness in both the magnitude and diverse sources 
of uncertainty in weapons acquisition. 3 


The operational definition of uncertainty was provided by 
the authors as "the relative unpredicatability of the outcome 
ef a contemplated action™. 

Herein lies a problem which has confronted those 
researchers who have attempted to subject the success of 
program outcomes to statistical analysis based upon a 
comparison of initial estimates and demonstrated performance. 
Peck and Scherer summarized the difficulty in this way. 


Let us begin by attempting to ascertain the degree to 
which the outcomes of weapons programs are predictable. 
A rough measure is obtained by examining three crucial 
kinds of weapons program outcome predictions: (1) qua- 
lity, or the expected technical performance and the 
reliability of the resulting weapon system; (2) develop- 
ment time, or the interval between the start of a 
development effort and the availability of operational 
weapons; and (3) the cost of development. Characteris- 
tically, predictions of these variables are made by 
contractors and/or buying agencies early in a weapons 
program and are recorded in written documents. The 
predictability of the variables can be determined by 
comparing the early predictions with actual quality, 
time and cost outcomes. 


Nevertheless, comparisons of this sort are properly 
viewed with considerable reservation, for there are 
extreme difficulties in obtaining satisfactory data. 
Neither the original estimates nor the final outcomes 
are completely comparable between programs, nor are 
they even necessarily representative of the true state 
of affairs within a given program. 


3 Peck, M. J. and Scherer, F. M., The Weapons Acquisition 
Process: An Economic Analysis, p. 17, Research Division, 


Graduate School of Business Administration, Harvard University, 
1962 


2 Tpid., p. 19 
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The reasons given did overlap to an extent with those 
reported by Marshall and Meckling. 

1. Initial predictions were made at varying points 

in the maturity of different programs, 

2. Initial estimates were optimistically low and 

in some cases deliberately inaccurate. 

3. Performance outcomes could not be measured 

quantitatively. 

4. Time of availability outcomes were qualitatively 

assessed. 

5. Cost outcomes were blurred by arcane bookkeeping 

procedures, whereby developmental costs were buried 

in production expenditures or post-production support. 
In a criticism of the Selected Acquisition Report (SAR), the 
Blue Ribbon Défense panel questioned the validity of before- 
and-after comparisons futher. 


The basic approach to the SAR is the establishment of 

a baseline of estimated costs, schedules and technical 
performance, and the subsequent measurement of the 
present status against this baseline. . .Successfully 
predicting the course of development of a new weapon 
system 1S uncertain at best. The long period of time 
involved introduces unpredictable changes as outside 
events and circumstances shift during the five to nine 
years it usually takes to acquire a new weapon. The 
development process itself contains hidden unknowns. 
The original estimates of cost, schedule and technical 
performance of a weapon system can be made with con- 
Siderable skill and with total honesty, but they remain 
only estimates, the worth of which can be determined 
only by the future unfolding of events. The SAR's 

tend to treat the original estimates as accurate pre- 
dictions and to measure subsequent events in the dev- 
elopment against the standard of the original estimates. 
There are two serious consequences of this procedure. 


WZ 





...-The overwhelming concentration now appears to be 

On maintenance of the costs and schedule within the 
Original estimate. Concern with the quality of the 
weapon and its ability to perform an essential mission 
are not presented in the SAR. . .Inhibition against 
change is the second serious consequence of the present 
SAR system. > 


The picture is not so bleak as one would be led to 
believe. Peck and Scherer pointed out the other side of 
the ledger as well in their analysis of the processes of 
weapons development. 


To examine the effectiveness of weapons program execution 
in an operational way requires a meaningful criterion of 
effectiveness. One possibility is to evaluate actual 
time, cost and quality results against original predic- 
tions. In particular, original time, cost and quality 
predictions do not necessarily reflect optimal choices. 
Still, the analysis of variances from original predic- 
tions has several values. For one, it is useful method 
of pinpointing the problems which affect program out- 
comes. Furthermore, to the extent that major national 
security decisions are made on the basis of original 
predictions, sizeable variances from those predictions 
may imply a serious misallocation of resources. 


A greater implication of the position that comparative studies 
are of no utility is that without comparisons, the effective- 
ness of eaSrnveg activities within the weapon system 
acquisition process must be left entirely to intuition. 

Peck and Scherer did attempt to pinpoint problems 
affecting program outcomes by correlating Marshall and Meck- 


ling's criteria of program success with a largely qualitative 


5 Blue Ribbon Defense Panel, Report to the President and 


the Secretary of Defense on the Department of Defense, Vol. l, 
p. 125, U.S. Government Printing Office, 1970 


6 Peck, M. J. and Sherer, F. M., The Weapons Acquisition 
Process: An Economic Analysis, p. 428, Research Division, 


Graduate School of Business Administration, Harvard University, 
1962 
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collection of explanatory variables. The results were 
characterized by a lack of statistical significance which 
Might be explained by a reliance on parametric procedures. 
However, the results did indicate that definite relation- 
ships existed between program characteristics externally 
imposed on the project, such as urgency, the importance of 
minimizing costs, state-of-the-art exploitation and program 
results. ’ 

This study was conceived in part as a redirected con- 
tinuation of the above aspect of Peck and Scherer's work; 
redirected in the sense that the choice of independent 
variables was limited to quantitative measures of a program 
(e.g., unit cost, total program cost, development expendi- 
tures and program length) which were then applied to a 
larger sample of programs uSing nonparametric analyses. 
However, more importantly this thesis has attempted to 
ascertain the validity of certain general beliefs of the 
nature of program management and its effect on program out- 
comes. Four separate hypotheses have been presented, each 
in its own right questioning the predictability of project 
Success when considered as the result of a single determinant 
variable.8 In short, the thesis has erected a model of the 


acquisition process in which the management skills and 


7 Peck, M. J. and Sherer, F. M,, The Weapons Acguisition 
Process: An Economic Analysis, ch. , Research Division, 


Graduate School of Business Administration, Harvard University, 
9162 


8 A fifth hypothesis was posed to test the data source 
fee bias’. 
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practices ef a system's program office staff or individual 
Manager have been ignored as predictors of program results. 
The model was further refined by suppressing the externally 
imposed noise of program contractions (expansions) and 
economic escalation. The hypothesis testing served to 
illustrate one dimensional differences in acquisition programs 
(i.e., cost growth differences in programs differentiated 

by project cost alone). 

In addition, a series of correlation tests were conducted 
in which the fundamental differences in program outcomes were 
further tested by controlling additional program character- 
istics such as service sponsor or weapon system type. The 
purpose of the correlation was twofold, first to establish 
whether the controlling process accented the gross differences 
identified by the hypothesis testing, and secondly, to identify 
among the program facets of size, length and risk, effective 


predictors of program outcomes. 


iS 





If. METHODOLOGY 


Se. ocebeCclTED ACQUISITION REPORT 

Recognizing the problem of how program maturity may 
distort the accuracy of early forecasts, it was considered 
essential that some instrument adjust initial estimates for 
all programs to a common beginning, and also to allow for the 
adjustment of these forecasts in the light of a lengthy 
program life. Such an instrument was the Selected Acquisition 
Report (SAR). 

The SAR serves aS a comprehensive, summary acquisition 
status report for the "highest levels of management within 
the Department of Defense and (is) the basis for reports to 
the Congress and other government agencies including the 
Government Accounting Office (enoy.*.,? Compiled at the close 
of each quarter, the SAR tracks the success of each major 
weapons system acquisition in the areas of cost, technical 
performance and schedule. 

In each of these areas, the report offers a comparison 
between current estimates for program success and the develop- 
ment, or initial, estimates. Sandwiched between these two 
expectations is yet a third, the approved program estimate 
Thieh @h practiveswvaries: only slightly, if at all, £xem the 


development estimate. The development estimate is established 


9 Secretary of the Navy Instruction 7000.3 
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once the Secretary of Defense grants his approval for the 
program to move into full scale development, and generally 
has aS its basis the Decision Coordinating Paper (DCP). 
The development estimate for procurement may be reviewed once 
the production contract has been let, but otherwise, the 
estimate may not be changed without permission of the Assis- 
tant Secretary of Defense (Comptroller). Fox points out 
the necessity of this last requirement as he quotes a DOD 
review of management control practices. 
This problem has been common enough to acquire the 
esoteric but descriptive label, “the rubber baseline". 
Translated, this means that the contractor managers 
would periodically make retroactive adjustments to 
cost estimates and schedules so that plans would 
equal actuals. . .Obscuring overruns in this fashion 
Centributedeto our being caught short on funding for 
two major programs last fiscal year. 
The approved program estimate is also based upon the DCP, or 
in its absence, a Program Memorandum or Program Change 
Decision. As these formal statements of changes in program 
goals are forthcoming, the estimate will necessarily be 
revised. The current estimate represents the program 
manager's most realistic assessment of program status. As 
the program matures, this appraisal will more and more equate 
to demonstrated performance or the actual completion of 
schedule milestones. The 31 March 1976 SAR for the A-7E 


project is provided as an example in Appendix A. 


The 30 June 1976 SAR was the principal, primary reference 


10 Fox, J. R., Arming America, p. 413, Harvard University 
Press, 1974 
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source of project data for this thesis. In fact, because of 
the completeness of the report and the clarity of its format, 
whether or not an acquisition program was included under the 
reporting system determined whether that program would be 
included in the analysis. Thus, this determination excluded 
a number of weapon systems presently in either development or 
procurement, principally in the areas of electronic counter- 


measures and naval gunnery. 


B. ANALYTIC METHODS 


1. ‘Hypothesis Testing 
}.™ Lat®ge programs have a tendency to incur greater 
Geet age Scueaule growth in absolute and relative terms than 
do smaller programs. 


One of the initial, underlying assumptions of this thesis was 
that not only will weapon system acquisitions expeMience cost 
growth, but that for system with large program costs, the 
growth will be correspondingly greater than for those less 
expensive projects. Reasons for the explanation of the cost 
growth phenomenon have been postulated again and again by 
critics of military procurement (e.g., Peck and Sherer, Fox). 
Not the least’ frequently observed reason is that of over- 
optimism. It would seem reasonable that large, expensive 
programs would require a certain amount of "selling" on the 
part of sponsors to ease the difficulties of higher level 

DOD and Congressional approval. This “selling” often took 


the form of understating the eventual program cost, and with 
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it the technological risk involved in the development, not 
to mention the date on which the system enter service. 
Whether overoptimism was the result of deliberate policies 
on the part of the sponsoring agency is something of a moot 
point. Peck and Scherer, however, did point out that the 
practice has been accepted as a matter of fact within the 
defense industry. 

Quite commonly, this strategy includes submitting 

proposals which in one or more aspects are highly 

optimistic. Indeed, the propensity to 'but into' 

attractive new programs with optimistic quality, 

time, and/or cost estimates is perhaps as much as 

an industry practice in advanced weapons acquisition 

as Ist price cutting is in automobile retailing, or 

as the SS ee of loss leaders is in department 

store operations.l 

Concurrent with the overoptimism that acompanies large 
programs, it was presumed that there would be difficulties 
encountered in the area of cost control simple because of 
program size, an extension of the belief that with size one 
inherits waste and a certain amount of uncontrollability. 
Unfortunately, it is impossible to distinguish explicitly 
between optimism or waste based on data gathered from the 
SAR" 's:. 

2. Programs with a larger development phase budget 

Hove avgneater tendency” to achieve initial cost, 


performance and schedule estimates. 


An acedemic approach to the study of project management has 


stressed the importance of "front-end loading" the acquisition 


jal Peck, M. J. and Scherer, F. M., The Weapons Acquisition 
Process: An Economic Analysis, p. 413, Research Division, 


Graduate School of Business Administration, Harvard University, 
1962 
19 





effort, citing the lever effect which early commitment of funds 
has upon program performance. l2 It would appear reasonable 
that programs which enjoy an extensive development period 
would suffer the least uncertainty in the expected level of 
program outcome, Since a greater percentage of the "unknowns" 
which plague any program would be uncovered. 


3. Programs with greater technical risk tend to incur 


greater cost and schedule growth than do less risky 
programs. 


This hypothesis is associated with the second contention in 
that programs which represent a great deal of technical risk, 
which require a more complex and sophisticated technical 
effort (as perceived from the outset of the program), would 


be more susceptible to failure. 


4. Multiple=-source contracted programs enjoy an advan- 
tage in program success over sole-source contracted 
programs. 


For the purpose of this analysis, those programs which had 
more than one prime contractor regardless of whether the 
Systems were in procurement or development were considered 
multiple source programs. In the case of air-craft systems, 
if there were competing contractors for either the airframe 
or engine designs, the program was considered to have a 
"second source". Frederick Sherer in his study of the 


economic incentive in the weapons system acquisition process 


12 the effect was originally suggested to this student 
by Professor Melvin Kline, U. S. Naval Postgraduate School. 
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documented the significant cost reductions which developed 
during the joint production of military aircraft in World 
War II. Although not in as much detail as Scherer, Fox 
touched upon the benefits of "Second sourcing", reporting 
that "it would be reasonable to assume a 25% reduction in 
contract price can be achieved in many, if not most, cases 
when competition is introduced."13 It would appear that 
this is one area in which weapons procurement responds 

to traditional market forces. The fourth hypothesis seeks 
to determine first, whether the evident cost successes do 
exist in a significant number of cases, and second, whether 
the advantage of multiple-source contracting is extended 


to the areas of performance and schedule. 


S- eft puospeet.of puoguam. successadimenmshes with 


increasing maturity among programs. 
This last hypothesis was posed in order to test the validity 


of the SAR in overcoming the criticisms alluded to earlier 
about the noncomparability of programs in differing stages 

of their life-cycle. For example, early in the life of a 
program, before the realities of the management or technical 
task at hand are more apparent, one would expect that current 
estimates of project outcomes would agree with initial 
assessments. But, as the program achieves greater maturity, 


and uncertainties arise, the gap between the two expectations 


13 Fox, J. R., Arming America, p. 256, Harvard University 
Press, 1974 
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would be sure to widen. 

Each hypothesis was tested using a one-tailed Mann- 
Whitney U Test (a nonparametric one way analyses of variance) 
in which the means of program outcomes were compared. For 
the most part, the sample of 48 programs would be divided 
along the median of the program variable in question, except 
in the case of the fifth hypothesis where whether a program 
was in development or procurement became the classifying 
criterion. The results of these tests are presented in 
Ghapter Iil. 

2. Correlation 

As stated earlier, associated with the hypothesis testing 
was a desire to examine the correlation between the output 
variables of cost, performance and schedule, and initial 
program estimates of program length, cost and technical risk 
when specific program characteristics were controlled for 
explicitly. The thesis employed a nonparametric correlation 
proceudre offered as a subprogram in SPSS: Statistical 
Pameage .20rthe Social Sciences by Norman H. Nie, et al. 
The following characteristics were considered and controlled 
for in deriving correlations between program outcomes and 
initial estimates: 

1. All forty-eight programs together without any 

differentiation; 

2. Programs differentiated by the maturity of their 


design (1.e., whether the program was in development or 
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had advanced to procurement) ; 

4. Programs differentiated by system type, ship, 
aircraft, missile or miscellaneous (tracked vehicles, 
fire~control systems, etc.); 

5. Programs differentiated by service sponsor, Navy, 
At’y, Or Alr Force. a 

The first category of ‘undifferentiated programs' was 
included as a control feature to Serve as a means of com- 
parison for the following four program characteristics. 

Items two and three were included as part of follow-up tests 
to the verification of the fourth and fifth bypotheses.,. 

The last two program characteristics, system type and service 
sponsor, were included because they are frequently used as 
means of ordering weapons systems. Also, there are sub- 
stantial inter-service differences in system acquisition, 

and as a result, there are natural, virtually instinctive 
inclinations to make comparisons between the programs of 

each service. 

Additionally, the intercorrelations among program out- 
comes were examined to see if any light could be shed on the 
trade-offs between cost, performance and schedule. The 
correlation results are presented in Chapter III. 

C. SELECTION OF INDEPENDENT VARIABLES 
Ll. Program Cost 
For the purpose of the analysis, it was considered 


essential that each of the forty-eight programs be considered 
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On an equal footing, that is, considered from the same base 
year. In this way, comparability between programs on the 
basis of total program cost would be maintained. The base 
year selected was 1974 since all but two programs (added late 
in the thesis) predated that year. Each of the programs 
reported development estimate program costs in their own 
base year dollars, which in turn were converted to 1974 
dollars. The method of accomplishing this conversion was 
not very satisfactory, but had to suffice for lack of a 
Superior alternative. From a Direct Budget Plan (TOA) 
History for the Department of the Navy, an implicit price 
deflator for the fiscal year 1964 through 1977 was derived. 
The unsatisfactory aspects of this source are evident when 
one considers that the TOA history included nonprocurement 
appropriations such as OM&N as well as the procurement 
related appropriations, RDT&E, SCN, etc. The U. S. Bureau 
of Economic Analysis is in the process of determining 
deflators Boe a of the basic defense industries, but 
results will not be forthcoming until late 1977. 

A copy of the TOA history and an example of the manner 
im which the implicit price deflators for each of the 
program base years was derived is provided in Appendix B. 

2. Pseguanm Unit Cest 

Unit cost was selected as an independent variable 

on the assumption that program success (primarily in the 


area of cost) could not be adequately explained by program 


24 





Size alone. For example, a program line the A-7E in which 
the Navy iS procurring a proven aircraft design with up- 
dated avionics, and the F-14A program which represents a 
completely new weapon system, could have conceivably identical 
program costs provided sufficient quantities of the A-7E 
were purchased. However, the true nature of both programs 
would be obscured. Of the two, clearly the F-14A with a 
program unit cost of $16.8M (1974S) is the more ambitious 
undertaking, and the lesser unit cost of the A-7E, $4.1M 
in 1974 dollars, would reflect this. As in the case of 
— cost, development estimates of unit costs were 
translated into 1974 dollars. 
3. Bevelopment Investment 

The development investment for each program was 
derived as the ratio of the level of development spending 
to total program as foreseen at DSARC II. No judgement of 
the manner of (or purpose for) development spending was 
attempted. The variable was intended to serve as a com- 
plement to program unit cost, that 1s, a measure of the 
technological advance of the project. Some difficulties 
were encountered in those reports for nuclear powered 
men-of-war where propulsion system development costs 
attributed to both the Navy and the Energy Research Devel- 
opment Agency (ERDA) were not included in the cost section 
of the SAR's, but rather in a SAR addendum. Consequently, 


at first glance a project as large and complex as NIMITZ 
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(CVAN 68) appears to be an off-the-shelf product, despite 
the fact that her two reactor propulsion plant represents 
a first for the surface Navy and is hardly a trivial tech- 
nological step. Adjustments were made for cases such as 
this by adding the additional estimated development cost 
to both the level of development funding and the total 
program cost. 

*. Bmeogram Length 

Of all the explanatory variables, time was the most 

subjectively determined. Initial program length, measured 
ostensibly from the initiation of the project to an initial 
operational capability date (IOC) proved to be a bit 
troublesome to extract from the SAR's. Earlier researchers 
have encountered similar problems. 

Time of availability predictions are often made in 

terms of an ‘initial operational capability’ - a 

decidedly qualitative concept. Does one missile 

on a lonely launch pad constitute an ‘initial opera- 

tional capability'? Or do 30 missiles when their 

reliability in time of need is doubtful? 
Fortunately, the term IOC is a common one within DOD and in 
a large number of cases, those dates were reported explicitly. 
In the case of Navy shipbuilding programs, a simplifying 
rule was established whereby the date on which the lead ship 


of the entire class in question became available for 


14 Peek, M. J. and Scherer, F. M., The Weapons Acquisition 


Process: An Economic Analysis, p. 20, Research Division, 
Graduate School of Business Administration, Harvard University, 
1962 
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operational deployment was chosen arbitrarily as the "I0oc" 
date. There are obvious draw-backs to this method when the 
entire class is considered as any student of ship construc- 
tion can point out. Namely, the greatest schedule slippages 
often occur during follow-ship production. Still, the 
availability of the lead ship does represent the realization 
of a design, and in order to maintain some parity with other 
systems produced more rapidly, the consistency of the above 
method was conSidered essential. 

The other end-point of program length, the initiation 
of the project, was even more difficult to determine. In 
most cases, a date corresponding to the advent of the system 
life-cycle was selected as the starting point. More often 
than not, this date was the date of approval for the system 
requirements/characteristics. There was a distinct lack of 
uniformity noted among the various programs in the reporting 
of this schedule milestone. In a few cases programs were 
considered underway with an all inclusive ‘commenced 
engineering development' or ‘initiated validation phase’. 

DA SELECTION OF DEPENDENT VARIABLES 
1. Cost Growth Factor 
Fortunately, a recent revision of the SAR system 
directed that development and current cost estimates be 
reported in constant year dollars as well as in escalated 
figures. This change has made the pinpointing of true cost 


growth that much easier. It came as something of a shock 
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to discover that little or no provision for escalation was 
included in the development estimate of many programs. 
Where some accounting for escalation was made, invariably 
the inflation rate was unrealistically low. 

The office of the Assistant Secretary of Defense 
(Comptroller) publishes a "SAR Program Acquisition Cost 
Summary" (see Appendix C) once each quarter's collection of 
individual reports is complete. From this summary it is 
possible to determine each program's cost growth both in 
dollars and as a percentage of original estimates. In each 
instance, the resultant figures are adjusted by OSD analysts 
to negate the effects of inflation and program quantity 
changes. Unfortunately, the relative measure of program 
cost growth is related definitionally to the original 
program cost éstimate as shown below. 

Cost growth (%) = Cost Changes 
Development Estimate + Quantity Changes 


(All entries are in base 
year dollars) 


One would expect a negative correlation between cost growth 
(%) and program size, expressed above as the 'development 
estimate'. The extent of the correlation would depend on the 
degree to which the Cost and Quantity Changes varied with 
total program “cost. TO aid in”“clarifying the effect oEL 
program size on cost success, it would be helpful to have the 


assistance of an absolute measure of growth, namely Cost 
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Growth ($) which is the dollar value of the overrun. This 
measure is determined as shown below. 
Cost Gremth ($) = Cost Growth(%) -l x Development 
Estimate 
Hence, a negative "growth" would indicate an underrun. 
2. Performance Achievement Factor 
Navy and Air Force concepts of system effectiveness 
were presented in the 1960's as a measure of how well a 
system performed its intended function. This figure of 
merit was derived originally as a companior to cost effective- 
ness in the hope that together the two would serve to deter- 
Mine whether a system undergoing design would satisfy most 
efficiently those operational requirements which spawned it 
in the first place. Despite interservice differences. in the 
wording of what was meant by system effectiveness, the 
measure (expressed as a probability) was generally conceeded 
to have three basic components; 
1. the performance capability of the system; 
2. the @perationgl readaness or avatilability 
that is, its ability to start performance of a 
mission when called upon to do so; 
3. the continued capability of the system to 
perform, or its mission reliability or dependa- 
preity. 
In time however, it became apparent that models of system 


effectiveness were not as useful as originally conceived. 
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The computations were messy and highly subjective. For 
example, how much does the failure of a single component 
degrade the effectiveness of a system? Or even more basic, 
what constitutes a failure? Since the derived probabilities 
were by no means “hard numbers", they were mistrusted by 
managers, and the computational seeeieiae of system effective- 
ness fell into disuse. 

Given that an evaluation of system performance is essential 
to any analysis of program success, and in the absense of a 
readily available performance factor counterpart to cost 
growth, a simplistic method of marking performance using the 
three basic elements of system effectiveness where possible 
was devised. From SAR data on the technical/operational 
CWarpacteristics of the system in question, it was possible 
to contrast current estimates of system performance with 
development estimates. Generally speaking, the performance 
achievement factor was determined by assessing a weighted 
average of major performance areas such as speed, time-on- 
station, maintainability, system accuracy, etc. A value of 
performance less than 1.0 indicates that the system achieved 
better results than originally estimated. An example of how 
this computation was made is provided in Appendix C. 

3. Schedule Overrun Factor 

As in the case of attaining performance goals, the 

schedule overrun factor was determined by the ratio of 


current to development estimates of program length. However, 
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like cost growth (%), shcedule slippage (%) is related 
inversely to program length. 
Schedule slippage(%) = Actual (Current Estimate Of) 
Program Length 
Development Estimate of 

Program Length 
Hence, a negative correlation would be expected. To minimize 
this dependency, a fifth output variable, schedule slippage 
(months) was introduced. Ina similar fashion as cost 


growth (S$), this variable measures the actual slippage of the 


TOC data in months. 
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I1ll. RESULTS 


= HESPORICAB PERSPECTIVE 

Appendix C has listed those programs from which program 
data was obtained for the analysis. With the exception of 
POSEIDON, all the programs are currently included under the 
SAR system. (POSEIDON reports were terminated in June 1975 
following completion of system deployment and the subsequent 
disestablishment of the program office). For the convenience 
of the reader, the programs are ordered by service sponsor. 
An examination of the outcome means for each service indicates 
that perhaps Navy programs enjoyed a greater achieved per- 
formance at greater cost and shcedule growth than did Army 
and Air Force projects. A sample comparison between outcome 
means revealed that the NAVY - USAF differences in all three 
categories were significant, while the pertinent ARMY - NAVY 
disparity was limited to the area of cost alone. 

The results of the NAVY - USAF comparison are not fully 
understood. Sample comparisons of Navy programs (less ship 
systems) and the Air Force projects shown did result in less 
significant cost differences. Additionally, when missile 
systems (which historically have proven to be technically 
more successful than other system types) were excluded from 
both samples, the performance difference between services 
was Still evident. No explanation for this is readily 
apparent, nor was there any system type peculiar to a 


Single service that could account for the Air Force advantage 
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in meeting deployment schedules 


Be HeeOrneols TESTING 


ee Large programs have a tendency Co Imcur Gredeer Gost 


and schedule growth than do smaller programs. 


Table 3-1 displays the difference in the means of 
program outcomes between large and small projects. The column 
marked ‘significance’ indicates the degree to which the 
differences were statistically significant. In instances of 
no significance, the column entry is NS. Subsequent tables 


in this section are arranged similarly, 


FABLE 3°— 1 


COMPARISON OF PROGRAMS BASED ON PROGRAM SIZE 


LARGER PROGRAMS SIGNIFICANCE SMALLER PROGRAMS 

Cost Growth(%) 1.13 0.003 Cost Growth(%) 1.29 

Performance 0.99 NS Performance 1.00 

Schedule Slip 1.16 On4005 Schedule Slip 1.44 

(3) (3) 

Schedule Slip 12.6 02005 Schedule Slip 25.2 
(Mos. ) (Mos. ) 

Eest Growth ($)534.0M 0.006 Cost Growth($)163.7M 


The results appear to provide mixed support for the first 
hypothesis. Ignoring the differences in relative cost growth 
for reasons given in Chapter II, note the differences in 
schedule overrun. Historically, larger programs have done 


significantly better in meeting original schedules, both in 
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a relative and absolute sense. 


The lack of significance 


observed in the area of achieved performance is a result 


that will be repeated elsewhere in this chapter. 


Overall, 


the nonparametric analyses of variance were ineffectual in 


accounting for the variance in system performance. 


In the area of cost growth($), 


appear to hold the line against overruns. 


larger programs’*did not 


The reasons for 


this may have included a tendency for the managers of large 


(and therefore important) programs to throw money at 


problems, particularly schedule problems, until a solution 


28 found. 


Once a major program has been funded, rarely 


has it had to do without additional resources if required. 


2. Programs with more development funding have a greater 
tencency to aghiieve” initial cos, yp perrommance same 


schedule estimates 


TABLE 3 - 2 


COMPARISON OF PROGRAMS ON BASIS OF LEVEL 


OF DEVELOPMENT FUNDING 


HIGHER PROGRAMS 
Geese Groweh (%) 1.15 
Performance 0.99 


Schedule Slip Lae 
(% ) 


Schedule Slip 15.94 

(Mos. ) 
Cost Growth 459.9M 
(S$) 


SIGNIFICANCE 
0.040 
NS 


OmMGOZ2 


NS 


NS 
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LOWER PROGRAMS 
Gost Growth(s) 1227 
Performance Leo 


Schedule Slip lee4ai2 
(3) 


Schedule Slip Ze 
(Mos.) 


Cost Growth Plage yan | 


(S$) 


Perhaps the most important comparison in Table 3 - 2 is that 
which contrasts relative cost growth. During the aneavene 

of program data, a very high (0.750-1.000) positive corre- 
lation was found to exist between program cost and develop- 
ment funding in a Significant number of cases. It was likely 
that the two were meaSuring essentially the same character- 
istic, program size. Thus, one would expect Tables 3 - l and 
3 - 2 to bear a resemblance to one another. In terms of 
relative cost growth and schedule slippage, this similarity 
exists. What is crucial about the results of this second 
sample comparison is that the level of development funding 
has a Significant negative impact on relative cost growth. 
Bearing in mind that the development cost is also an indirect 
measure of program size, this is the first instance of an 
indication, not confounded by definitions of the variables, 
that suggests program size may inhibit the degree of cost 
@rewth. Further signs of the efficacy of early and sub- 
stantial investments are the differences in relative schedule 
slippage and the absence of any significance to the difference 


between the absolute cost growth for each sample. 


3. PBeogsams Wane) -peuce ved technaeal risks tend te aneur 
greater cost and schedule growth than do less risky 
programs 
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TABLE 3 - 3 


COMPARISON OF PROGRAMS ON 
BASIS OF DEVELOPMENT INVESTMENT 


HIGHER PROGRAMS SIGNIFICANCE LOWER PROGRAMS 

Cost Growth(%) 1.18 NS Cost Growth(%) 1.24 

Performance 0.97 NS Performance 0.97 

Schedule Slip las 2 NS Schedule Slip iz33 
(3) (3) 

Sehedulie Slip 19.4 NS Schedule Slip ise 2 
(Mos. ) (Mos.) 

Cost Growth($) 349.8M NS Cost Growth($) 340.3M 


The weakness of development investment as a explanatory 
variable is apparent in Table 3 - 3. No significance can 

Be aetrmbuted tothe difference in oftcomes between programs 
with a higher development/committment and those with less. 

A second comparison between the extremes of each group, the 
highest and lowest quartiles, was made with similar null 


results. 


ae Whiltiple=seurce contractedPpregeans enjoy an advan- 
tage Bs program success over sole-source contracted 
programs 
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TABLE 3 = 4 


COMPARISON OF PROGRAMS ON BASIS OF 
PRODUCTION/DEVELOPMENT SOURCES 


SOLE=SOURCE SIGNIFICANCE MULTIPLE SOURCE 

Cost Growth(%) 1.21 0.040 Cost Growth(%) 1.14 

Performance 0.99 NS Performance 02 

Schedule Slip 1.30 NS Schedule Slip 1.44 
(%) (3) 

Semegule Slip 17.8 NS Schedule Slip 24.7 
(Mos. ) (Mos. ) 

Cost Growth($) 382.3M 0.049 - Cost Growth($) 103.4M 


The findings of Peck and Scherer were sustained by the Mann- 
Whitney test. In terms of relative and absolute cost growth, 
second-source programs did better than their sole-source 
counterparts. ~ Although a small class of projects (7), the 
multiple source programs included Army and Navy projects of 
all system types. In regard to the achieved performance of 
both program types, this study found no significance in the 
difference of program outcomes. This 1s at variance with the 
findings of Rubin and Marquis' who determined that "sole source 
contractors do significantly better technically than contrac- 
tors who were selected by some form of competitive procedure. "16 


Rubin and Marquis' study 1S not precisely comparable to the 


16 Sloan, Alfred P., School of Management, Massachussetts 
Institute of Technology, Report #192-66, Critical Decisions in 


the initiation of Development Projects, Rubin, 1. Mosand 


Marquis' June 1966 
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one undertaken here in that only development projects in 
which the technical performance was evaluated subjectively 
by management and engineering personnel, were considered. 
However, the contention that greater technical performance 
can be achieved in the absence of competition was not 


supported. 


5. Thewprespect of program success diminishes with 
Iinmereasing maturity among programs 


TABLE 3 - 5(A) 


COMPARISON OF PROGRAMS BASED 
ON LIVE=CYCLE DIFFERENCES 


PROGRAMS IN PROCUREMENT SIGNIFICANCE PROGRAMS IN DEVELOPMENT 


GCest Growth(%) 1.22 NS Cost Growth(%) 1.18 

Performance 0.98 NS Performance ° OO 

Schedule Slip 1.33 NS Schedule Slip 1.29 
(%) (3) 

Schedule Slip Loa7 NS Schedule Slip SS 
(Mos.) -° (Mos.) 

Cost Growth($) 460.6M Ge 02 2 Cost Growth($) 213.9M 


Table 3-5(a) reveals that on a gross level the SAR is able to 
minimize the differences among programs that arise because 

of life cycle disparities. With the exception of program 
growth in dollars, programs in procurement and those still in 
development exhibited similar success. The exception of 
program growth is indicative that either uncertainties in 


schedule or producibility do still exist, and that the 
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development phase has not been completely successful in 
identifying and costing these aspects of the system. It is 
felt that the lack of significance between the two categor- 
ies can be attributed to allowing the SAR development 
estimate to be reviewed and updated when required once a 
production contact has been let. 

In the event that differences within groups were obscured 
by combination, the development and procurement classes were 
again divided, in the development cast by the age of the 
program, and in the case of procurement by whether the system 
had reached I0C. These two groups were then compared and 
the results presented in Table 3-5(b). 

TABLE 3 - 5(B) 
COMPARISON OF PROGRAMS BASED ON 


a LEFE=CYCLE DIFP ERENCES 


PROGRAMS AT/PAST IOC SIGNIFICANCE RECENTLY INITIATED PROGRAMS 


Cost Growth(%): 1.22 NS Cost Growth (3%) eeslaG 

Performance Oee9'6 0. 02S Performance 0.99 

Schedule Slip Lew29 NS Schedule Slip 135 
(3) (3) 

Schedule Slip 16.4 NS Schedule Slip 24.1 
(Mos. ) (Mos. ) 


Geee Growth($) 523.8M 0.050 Cost Growth($) 266.5M 


The results were generally supportive of those shown in 
Table 3 - 5(a). Once again the difference in cost growth 


is significant; however, the difference in the means of 
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achieved performance was unexpected. One possible explana- 
tion is that the realities of a fully operational deployment 
might test system capabilities better than a formal evalua- 
tion period, principally in the important areas of relia- 
bility and maintainability. On balance, the findings in 
Tables 3 - 5(a) and 3 - 5(b) supported the contention that 
the SAR in its present form does overcome sufficiently the 
difficulties in comparing programs at various stages of 


completeness. 


&. CORRELATION 

The results of the nonparametric correlation are presented 
in Tables 3 - 6 through 3 - 10(c). Overall, the results are 
not impressive. To be sure, there were individual cases in 
which the controlled characteristic produced effective 
results. For example, missile programs displayed an across- 
the-board effectiveness in predicting success. (See Table 
3 - 7(b)). Why this is true of missile systems and not ships 
is not readily apparent. For whatever reason, there is a 
greater homogeneity among missile programs than among other 
groupings. Perhaps then, one could suggest that greater 
faith could be placed in the initial estimates of missile 
program success to the extent that they are valid indicators 
of future results. Characteristic, however, of the analysis 
was the finding of less than 50% of the correlations as 
Significant. And, when those variable pairs with definitional 


relationships are discounted, the results are even more 
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nebulous. Additionally, the whole purpose of ordering 
systems by some specialized criterion variable to isolate 
the effects of program determinants is called into question 
when one recognizes that the category of “all programs" 
exhibited a greater number of significant correlations than 
a majority of the other instances when a program character- 
istic, like service sponsor, was controlled. Actually, the 
taxonomic analysis was weakened further by disparities in 
sample size among some of the classes of programs. The 
greatest difference was noted between the sole-source 
contracted programs (40) and multiple-source contracted 
programs (7). It is not unexpected that those categories 
had by comparison more significant results than their smaller . 
counterparts, Since greater sample size lowers the level at 
which a relationship between variables can be deemed signifi- 


cant. 
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TaeLeE 3 -— 6% 


ALL PROGRAMS 


Performance Cost Cost Schedule Schedule 


Factor Growth Growth Sip Slap 
Factor Factor Factor Factor 
(3) (S$) (%) (Mos.) 
Program ; 
Ureaet 0.1193 -0.2611 0.1430 +0.1668 -0.0682 
Cost (0.038) 
Program Oeil -0.3488 0.4549 -0.5408 -0.3892 
Cost (OVO00G) (0.002) (0.001) (0.003) 
Program 
Development 0.0545 -0.1415 -0.0964 -0.0746 -0.0608 
Investment 
Program 0.2622 -0.2763 0.0999 +-0.3898 -0.0664 
Length (0.027) COR US, (0.004) 


* Entries in parentheses indicate 
level to which Spearman correla- 
tion is significant. All others 
are not significant. 


Sample Size 48 


Program Outcomes (Mean) 
Performance 0.992 
Cost Growth (3%) L220 


Schedule Slip (%) i320 
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Unit Cost 


Votal Prograina 
"Gis 


Deve lopment 
Investnent 


Procrama Lenauth 


sample S1ze 
fe) 


wire Cost 


“otal Prodrani 


Cost 


Levelopment 
Investment 


Pro.rain Lendth 


wanple .1Ze 


Eo 


16 


cltiedule 
SUL ome eA) 


0.0678 


=U 7 
(0.035) 


0.4120 


~0.1362 


wuchednle 
J ueeathose) 


0.5439 
(0.0605) 


-0.494 
(0.088) 


0.5297 
(6.071) 


0.3613 


eee Sas a) 
io 
ple Se 
Performance frost (fost 
achievenent ‘rrowth (.) Growth (‘p) 
N/A -(}.3766 -U.S0v7 
(9.056) 
N/A -JU.7488 -0.4333 
(0.011) 
N/A 0.0890 ~0.2447 
N/A -0.3109 “0.4043 
Mean Outcomes 
Performance Cost Growth 
L.QUG rare. 
Psa es 
‘oS. Yea 
Perrormance Cost Cost 
Achievement Growth (%) Srowth (3) 
-0. 512 -0.3305 Of255'S 
(OeOso) 
=) 01.23 ~0.4677 O60 7 
(0.0275) (0.004) 
OF04 32 «O\557 16 -O.1461 
(0.007) 
O47 | ~-0.2992 0.4363 
78672 5.) (0.040) 


Performance 


igoy 


ean Outcomes 


6s tecnowt oO 


Ieee 


Senedule Slip 
1.3908 


uchedule Schedule 
Stitt.) ( Sinipattose) 
-~0.4653 -0.3940 
(0.022) (0.048) 
-0.60839 ~U.5 70 
(O,0CT) (9.006 ) 
-(3.3478 ~0.4118 

(CO x072) (0.040) 
-0.7703 -0.4780 
‘O,0GT) (Os022) 


schedule iin 
lasus 


Wo correlations with achieved performance were possible because each 
each of the programs in the sample was assigned a factor of 1.0. 
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Unit Cost 


Total Program 
Cost 


Deve lopment 
Investment 


Program Length 


Sample Size 
14 


nit Cost 


Total Program 
Cest 


Deve lopment 
Investment 


Program Length 


Sample Size 
6 


TABLE S a TC) 


AISeRAYT Sissi 


Pertormance Cost 
Achievement Growth (%) 
-0.922 0.1694 
0.3709 -0.16065 
0.1594 0.2381 
072001 -0.2090 
Performance 
0.991 
- TABLE 3 -~ 


Cost 
Growth ($) 


0.3978 
(0/5078) 


0.9341 


(QnG25) 


0.0991 


-0.3055 


Mean Outcomes 


Cost Growth 


1.149 


7(D) 


MISCELLANEOUS SYSTEMS 


Performance Cost 
Achievement Growth (%) 
0.0 (ie 2000 
-0.1160 -0.0286 
0.2399 -0.0857 
-0.2500 0.2029 

Performance 
0.998 
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Cost 
Growth (35) 


0.9429 


1.0000 
(0.001) 


0.2971 


0.2899 


Mean Outcomes 


Cost Growth 


ie? © 


Schedule 
Sivors) 


0.1046 


-0.1246 


0.5062 
(0.032) 


-0.1357 


Schedule 
slip (%) 


-0.5000 


-0.8000 
(02052) 


-0.3000 


-0.6667 


Schedule 
Slip (Mos. } 


0.0871 


-0.0380 


0.4289 
(07063) 


-0.0112 


Schedule Slip 


1.149 


Schedule 
ole (Mos.) 


-Q.1000 


-0.6000 


0.1000 


-0.3591 


Schecule Slip 


1.170 


TABS 8. = ofa 


SOLE@#SOURCE CONTRACTED PROGRAMS 


Performance Cost Cost Schedule Schedule 
Achievement Growth (%) Growth (S$) Slip (%) Slip (Mos.) 
Unit Cest 0.2601 -0.3363 Osi433 =OsrtO3s 0.0243 
(0.018) 

Cost (0.020) (0.001) COROOT ) (Oeon? ) 
Deve lopment 0.0392 -0.1106 -0.0598 ~0.0448 -0.0539 
Investment 
Prograr. Lenath 0.22888 -0.1480 0.2477 -0.2710 0.0464 

(G2 0387 ) (09067 ) (0.050) 
ean Outcomes 
Sample Size Performance Cost Growth Schedule Slip 
40 0.986 e207 Tigoyd (0)! 
; TABLE 3 - 8(8) 
MULTIPLE-SOURCE CONTRACTED PROGRAMS 
Performance Cost Cost Schedule Scheciule 
Achievement Growth (%) Growth (5) Slip (%) Slip (Mos.) 
Unik Coast -0.2041 -0.6429 =O%2G07 -0.5357 =Oaco7 i 
(0.060) 
Total Procram -0.6124 -0.4643 030257 -0.6429 -0.5714 

Cost (0.072) (0.060) (0.090) 
Develonmment 0.6124 =O72143 O7oo7 L -0.1786 -0.0714 
Investment (Os07 2) 

Program Length 0.0 -0.4140 -0.1261 -0.6847 -0.5225 
(0.045) 
Sample Size Performance Cost Growth Schedule Slip 
7 PaO23 1.140 1.443 
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Performance Cost Cosit Schedule Schedule 
Achievement Growth (%) Growth (3) Siting (5) Siaip Mos.) 
eat Cost 0.0901 ~0.4680 -0.2342 0.0462 Oster 
(0.008) (0.079) 
Total Program 0.0619 -0.3233 0.4188 -0.61398 -0.5926 
Cost (0.054) (0.017) (0.001) (0.001) 
Development -0.1763 -0.2361 -0.1229 -0.4241 =O0b7 5 
Investment (03012 ) 
Program Length 0.3805 -0.2419 -0.0589 -0.3964 -0.1602 
(0.028) (Os037 ) 

Sample Size Performance Cost Growth Schedule Slip 
27 0.976 Le2go leoos 
“ABLE 3 - 9(B) 

PROGRAMS IN DEVELOPMENT 

Performance Cost Cost Schedule Schedule 

Achievement Gréwth (%) Growth ($) Sip) Sips.) 
Unit Cost 0.32953 -0.1598 OgG2Z209 0.0270 0.0418 

(0.058) 
Toztal Program 0.3298 -0.4071 0.4414 ~0.4880 -0.0984 
Cost (0.098) (0.2) (0.033) (07023) 
Development ees -Q29606 0.0319 -051501 -0.0486 
Investment (O20 } (0.074) 
Progran Length O233d04 -0.3544 023790 “9.2931 OAS 
(0.098) (0.081) (0.067) 
Mean Outcomes 
Sample Size Performance Cost Growth Schedule Slip 
al: 0.9355 es e293 


TABLE 3 ~ 9(A) 


PROGRAMS IN PROCUREMENT 
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nit Cost 


Total Program 
Cost 


Development 
Investment 


Program Length 


Sample Size 
9 


Umit Cost 


Total Pro ram 
Cost 


Development 
Investment 


Program Length 


Sample Size 
14 


Themes 3 - 1008) 


NAVY PROGRAMS 


Performance Cost Cost Schedule Schedule 
Achievement Growth (%) Growth ($) Slip (%) Slip (Mos. ) 
0.3760 -0.5484 0.2130 oO AS -0.0185 
(09082 ) COeCC 3 } 
O.22:866 -0.6306 0.1304 -0.6177 «5 Jaz 
(0.082) (0.001) COeo0t } (0.001) 
0.0557 O £01337 -0.2331 -0.1079 -0 .0922 
Ove 31.7 ai. 52.7 3 -0.1628 -0.4471 -0 321837 
(0.016) (0.004) (0.013) 
Mean Outcomes 
Performance Cost Growth .'! Schedule Slip 
0.969 Ve? 2 1.404 
TABLE 3 - 10(B) 
ARMY PROGRALS 
Performance Cost Cost Schedule Schedule 
Achievement Grewth (%) Grémeh (S$) Slip (%) Slip (Moss) 
-0.0305 -0.2992 0.0374 -0.1923 O10 
-U .1997 -0.3036 0.5604 -0.5110 -0.0718 
(0.019) (0037 ) 
0.0113 -0.3297 -0.3128 0.0826 0.2188 
0.2744 -0.1506 -0.1084 =-0.6177 G20507 
(OeOie2) 
Mean Outcomes 
Performance Cost Growth Scredule Slip 
1,075 1.149 e292 
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TAREE 3 - 10(C) 
USAF PROGRAMS 


Performance Cost Cost Schedule Schedule 
Achievement Growin (5) Grouth (3) “Sivp (%) "Slip (Mos.) 


Umit Cost O21 S ero / 0.4167 -0.0848 OAS 
Total Progran O7eLss O(a 0.7300 -0 .2834 =) 2087 O 
Cost (02012) 
Levelopment -0.6659 >. O30 0.2678 0.1932 O.28e3 
Investment (0.027) 
Program Length Of2283 -0.0352 Owg2'93 -0.0479 0.4072 
(0.091) 
ifean Outcomes 
Sample Size Performance Cost Growth Schedule Slip 
9 1.006 1.141 Reale!) 
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Table 3 - 6 has displayed the correlation results for all 
Gerty-elight programs without any differentiation. The 
strength of program cost in determining schedule success is 
apparent by the significance levels of the correlations 
(0.001 and 0.003). Table 3 - 7 reveals the superiority of 
the missile system correlation over the other system types. 
Eleven out of eighteen valid correlations were significant 
in the case of missile programs, while ship systems produced 
but five. Oddly enough, aircraft systems with a larger sample 
Size than ship systems had only four significant correlations. 
Apparently, aircraft systems were responding to other deter- 
inants, most probably managerial in origin. 

Despite the great difference in Sample size note pre- 
viously, multi-source programs and sole-source programs pro- 
duced very nearly the same number of significant correlations. 
(See Table 3 - 8). In the case of the second source programs, 
the lack of impact on the part of program cost on absolute 
cost growth noted earlier was suprising, thereby suggesting 
that cost benefits do occur as the wsult of competition. In 
common with multi-source contracted programs, developing 
programs reflected a decreasing level of performance with 
increasing technical risk. This expected result was 

curiously not demonstrated elsewhere. The differences 
between service programs discussed earlier are exhibited in 
Table 3 - 10. Not suprisingly, with a larger sample size, 


Navy programs produced more significant correlations than 
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either of the other services' programs combined. 

Table 3 - ll represents a summary of the statistically 
Significant ( 0.100) tests presented in Tables 3 - 7 through 
3 ~ 10, showing the degree to which variable relationships 
were common regardless of the controlled program character- 
istic. Similarily, Table 3 - 12 displays a summary of the 
correlation (not shown) of dependent variables with them- 


selves and the degree of commonality encountered. 
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TABLE 3 


alla 


SUMMARY OF SIGNIFICANT CORRELATIONS BETWEEN 


DEPENDENT/INDEPENDENT VARIABLES 


Performance 


Ueoat Cost 

Progam Cost 
Development Investment 
Program Length 


Cost Growth (%) 


Unit Cost 

Program Cost 
Development Investment 
Program Length 


Schedule Slip (3%) 


Sanit Cost 

Program Cost 
Development Investment 
Program Length 


Schedule Slip (Mos.) 


Urmat Cost 

Program Cost 
Development Investment 
Program Length 


Cost Growth (S$) 


Unt Cost 

Program Cost 
Development Investment 
Program Length 


6 | 


Positive 
il 
9 
0 
4 


Positive 


OOO © 


Positive 


OV NM W bo 


Positive 


OrOO 


Positive 


J 
0 
2 
0 


Negative 


OOO WN 


Negative 


WON ~1 OV 


Negative 


OOoOr oO 


Negative 


1 
10 
1 
7 


Negative 


dL 
5 
1 
1 





TRBSE 3 - l2 


SUMMARY OF SIGNIFICANT CORRELATIONS BETWEEN 
DEPENDENT VARIABLES 


Performance Positive Negative 
Cost Growth (3%) 0 6 
Cost Growth (S$) 0 0 
Schedule Slip (%) 0 0 
Schedule Slip (Mos.) 0 0 

Cost Growth ($%) Positive Negative 
Cost Growth (S$) 10 0 
Schedule Slip (%) 9 0 
Schedule Slip (Mos.) 4 0 

Cost Growth ($) Positive Negative 
Schedule Slip (%) al 2 
Schedule Slip (Mos.) iE 2 

Schedule Slip (%) Positive Negative 
Schedule Slip (Mos.) 12 0 


- 


The results of the correlation analysis as summarized in 
the above tables illustrate some interesting points. The 
primo of ae the weakness of the independent variable, 
development investment. Construed as a measure of tech- 
nological risk, the variable failed to correlate with any of 
the outcomes except ina few cases. One of two conclusions 
can be reached, either the program charactersitic of risk 
is valid and the variable as defined did not capture it, or 
risk has little or no effect on program outcome. It 1s 


suspected that the former is true based upon the results 


HZ 





reported by Peck and Scherer.1/ Also, on reflection, it is 
felt that the data as presented in the SAR's could not be 
manipulated otherwise to measure risk. 

Much the same criticism can be directed toward the use 
of program unit cost. With the exception of relative cost 
growth, unit cost did little to explain program outcomes. 
The fact that unit cost did correlate significantly with 
relative cost growth in six out of a possible twelve cases 
was unexpected. The variable was intended initially to 
support development investment as a measure of the complex- 
ity of the program. Intuitively one would expect that the 
relationship between unit cost and cost growth (%) to be a 
positive one, not negative. It is suspected that unit cost 
may be reflective of an unknown project feature. 

From Table-3 - 12, a summary of the correlations paereen 
dependent variables reveals that a definite relationship 
between cost and schedule growth exists, perhaps more aptly 
put as Mwave is money". As in the case of the analysis 
involving both dependent and independent variables, perfor- 
mance achievement remained a largely unexplained outcome, 
although in 6 of ll trials, a Significant negative correla- 
tion between relative cost growth and Pe ee report- 
ed. It may be hasty to say that a desire to reach or surpass 


initial performance levels causes cost overruns since a 


17 Peck, M. J. and Scherer, F. M., The Weapons Acquisi- 
tion Process: An Economic Analysis, Chapter 16, Research 
Division, Graduate School of Business Administration, Harvard 
University, 1962 
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corresponding relationship between performance and cost 
growth in dollars was not evident. The overwhelming 
number of significant correlations between relative and 
absolute cost overruns, and schedule slippages must be 


discounted on definitional grounds. 


Se. CONCLUSIONS 
From the results of the hypothesis testing, three 


major conclusions have been derived. 


1. Contrary to previous belief, large, expensive 
programs exhibited a greater controllability in terms 
of schedule than did PEO Sees of smaller size. 


One possible explanation is that large, expensive pro- 
grams attract a great amount of attention from every quarter, 
the sponsoring service itself, OSD, Congress and the public. 
Conceivably, greater pressure are brought to bear to keep 
large programs within the limits of an original assessment 
as suggested .by the Blue Ribbon Defense eeweie and thereby 
avoid the embarassment of adverse program outcomes. A 
second reason (and one not necessarily exclusive of the 
first) might be that large programs, because of their size 
and sensitivity, are subject to greater scrutiny before the 
projects get underway, and thus the original estimates are 
more realistic. Also, once underway, program offices may 
find themselves more adequately staffed both in numbers of 


people and their expertise to ensure success. 
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2. Despite indications that early and substantial 
funding may reduce the scope of future cost growth, 
Phere is little clearcut effect of "front-end" money on 
the technical and operational outcomes of the program. 


Although no judgements were made of the manner in which 
development funds were spent, there are indications that 
the management of the development/system definition phase 
of the life-cycle should be redirected. The staff report on 
the acquisition process to the Blue Ribbon DefenSe Panel in 
1970 reported that the services had failed to receive 
sufficient funding to carry out necessary technical work 
during the early periods of their programs. What would often 
happen then was that critical engineering studies would be 
conducted during the production phase when money was more 
abundant. It is plausible that an adverse effect on system 
capabilities and performance would be suffered. 

Related to this last point is the complete lack of effect 
that development investment displayed when defined as a 
ratio of development funding to program cost. It may be 
entirely possible that the definition of development invest- 
ment is inadequate to the task of measuring technical 
uncertainty. Despite the fact that a majority of programs 
(40 out of 48) met or surpassed the initial performance goals 
established for them, and that the "Science" of predicting 
what is technically feasible may be more efficient than the 
"art" of estimating cost or schedule outcomes, one should 


be reluctant to surmise that developmental effort has less 


oe 





SRecee on pRojeet success than others=pregram variables. 
Although certainly less program resources are committed 

to advancing the state-of-the-art than manufacturing final 
hardware, judging from figures for the cost growth of 
strictly development budgets as reported by Peck and Scherer, 
cost variance has exceeded three or four times the original 
estimates in the past. It is conceivable that these develop- 
mental excesses seta have a snowballing effect of the 
remainder of the program, expecially on schedule milestones 


if stirct measures against concurrent development and pro- 


duction are taken. 


35 (‘Feoqmwan lemeth is related inversely. to the achieve 
ment goals. 


It appears from an examination of all the characteristics 
that performance estimates were either met or exceeded, and 
so it is surprising to find indications that program length 
is related in a positive way to achievement of desired 
performance. Peck and Scherer came across a possible explan- 
ation in their study. 


Each engineer wants to perfect things on which he 
works. Every development nevertheless involves 
compromises between quality: and time. If schedules 
are set too generously, there is a tendency for 
persons werking on the prog@ct to use up enersene-— 
duled time and still be no nearer completion on 
their hardware goals than under a shorter schedule. 
Therefore, in order to provide a good target a 
schedule should be short enough to make people think 
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and worry about getting the hardware out.18 
Unfortunately, program length contributed very little else to 


an understanding of program success. 


18 peck, M. J. and Scherer, F. M., The Weapons Acquisition 
Process: An Economic Process, p. 413, Research Division, 
Graduate School of Business Administration, Harvard University, 
1962 
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IV. COMMENTS AND RECOMMENDATIONS 


Fundamental to this thesis was the point of view from 
which the analysis was conducted. Each of the forty-eight 
programs were considered as "black boxes" into which certain 
inputs were made (cost, time, effort, etc.) and from which 
certain measurable outputs were collected. Meadows and 
Marquis criticized such "post mortem” techniques in their 
own analysis of the implications of forecasting errors for 
ignoring the interaction between forecasts and outcomes in 
the process of development.1? Additionally, Determinants 
Se Freyect Success by Murphy, et al, stated, “with rare 
exception, the determinants of project success were manage- 
Meme factors, things which management had the potential 
apa lect y to influence...",*9 a conclusion which necessarily 
required a aot of view from within the box. 

The analysis undertaken here does not mean to suggest 
that management techniques are without effect in determining 
program outcomes. Instead, the study asked how program 
initial conditions related to program outcomes, or more 


precisely, to what extent could the initial conditions be 





19 Meadows, D. L. and Marquis, D. G., Characteristics 
aud Gmolications of Forecasting Bunemgs in thelselectieniet 
R&D Projects, paper delivered by D. L. Meadows at The 
Second Annual Technology and Management Conference, Washing- 
Bem, D.C., 21 March 1968 


20 Morphy, D. C., Baker, B. Ne game Pacher, D., Peters 


minants of Project Success, p. 5, Management Institute, 
School of Management, Boston College, 1974 
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relied upon to "explain" program success. '‘'Explain' is 

used here in the sense of signalling the degree of success 
that a program most likely would enjoy. And because of 

that, Meadows and Marquis are correct, the transfer function 
e® the black box is not really of primary concern. Actually, 
the results of this study do uphold Murphy in that the 
weakness of the correlation implies that other determinants 
are at work. 

Before dismissing the methodology as indeciSive at best, 
perhaps a few suggestions as to how to improve it would be 
in order. During the accumulation of data for the thesis, 

a concerted effort was made to ascertain a measure of the 
relative importance of each project without success. The 
naw practice of assigning importance classification based 
on the significance of the preceived threat to program once 
the Specific Operational Requirement has been drafted is no 
longer in effect, and no joint ranking system appears to 
exist. The addition of this extra variable as a measure of 
how the program is perceived outside the project office 
would strengthen greatly the correlation prodedure and no 
doubt serve as the basis for an interesting hypothesis as 
well. 

It is felt that the data from the SAR should be supple- 
mented by information gained from other sources, such as 
interviews or questionaires. For instance, the dilemma as 


to what effect technological advance has on project success 
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could be settled by correlating the technical assessment of 
program office engineering personnel with the SAR measure- 
ment. Additionally, some understanding of how initial 
forecasts drive the program, that is how the attitude of 
managers toward these development estimates is reflected in 
the outcome, might prove an interesting variable in itself. 
Obviously though, the distance between the analyst and the 


programs would have to be shortened. 
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APPENDIX B 


CGMRBUTATION OF IMPLICIT PRICE DEF LATORS 


The Direct Budget Plan (TOA) History (following page) 
was the source of the implicit price deflators discussed 
earlier. The following example will demonstrate how these 


conversion factors were derived. 


A,- 7E Program 
From the SAR in Appendix A, the program base 


year is established as 1967. Then, from the TOA history, 
he Tatio of FY 1967 obligations to an equivalent amount of 
1977 dollars is computed. This is: 

$21,190M/$42,904M = 0.493 
Similarily, a ratio of FY 1974 obligations to 1977 dollars 
is also calculated. This is: | 

$26, 888M/S32,924Gm = 0.817 
The second ratio is then divided by the first and the result 
is a price deflator for converting 1967 dollar amounts to an 
equivalency in 1974 dollars. 


0.817/0.493 = 1,66 
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APPENDS 


COMPUTATION OF OUTCOME VARIABLES 


Listed in the following pages are the forty-eight programs 
which constituted the sample of acquisition programs studied 
and their associated outcomes. Before proceeding further, 
it was felt that some examples of how each of the outcome 
factors was obtained would be beneficial. The A-7E SAR in 


Appendix A will serve as a data source, 


1. Cost Growth (%) 

As stated previously in Chapter II, the measure of 
relative cost growth was obtained directly from “SAR Program 
Acquisition Cost Summary". That document reported the A-7E 
project relative cost growth as + 26%. Thus, the program has 
au@ost Growth factor of 1.26. | 

2. Performance 

In Chapter II, the achieved performance factor 1s 
stated as gerne derived as a weighted average of key per- 
formance criteria. In the case of the A-7E program, these 
were: 

1. maximum speed at sea level; 

2. xvadius/range during primary attack mission; 

3. ceLling/altiteude: 

4. reliabi lise, 

5. maintainability - 


a. Scheduled Depot Level Maintenance (SDLM) 
b. Maintenance Man-Hour per Flight Hour. 
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A comparison of the current estimate/demonstrated performance 
and the development estimate for each of the criteria above 
yields the following results. 

le SOSH KEBS/555 Kis: = Oggi 

2. 537 NM/475 NM = 1.13 

Se 3 SeebOOeft/3Se0000fE = 1.00 

4. Ooere/0ms4 sodmee 

5a. 24 MOS/30 MOS = Q.80 

b. 3.4 HR/I9e6 GRe= O456 


(Weighted equally, maintainability 
is 0.68) 


Summing all of the scores and dividing by the number of 
criteria, one achieves a performance factor of 0.93 for the 
program. 

The above procedure is illustrative of the general 
approach ~~ There were two modifications to this method, 
one in the case of ships, the other in the case of certain 
one-shot devices. Unlike aircraft where speed, maneuver- 
ability, range, reliability and maintainability taken toge- 
ther represent a meaningful measure of system performance, 

a comparison of SAR criteria for ships (e.g., speed, manning 
level, displacement, etc.) does not. Invariably these 
criteria are met. The existence of multiple mission areas for 
a single ship type further complicates the figure of merit 
calculations. This is a good example of where system 


evvectiveness breaks down in a computational sense. For these 
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reasons, ship systems were arbitrarily assigned a performance 
fa@etor of 1.0. 

One-shot systems (less strategic systems) were evaluated 
based on a comparison of their single shot probability of 
kill given in the SAR. This assumed the systems were activated 
within their proper search/attack envelopes and that all 
components functioned properly. 

3. Schedule Growth (3%) 

From the Schedule Milestone section of the SAR, the 
program was initiated in May 1963 and achieved an IOC on 
schedule in April 1970. Thus, there was no schedule slippage 


and a factor of 1.0 was awarded. 
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PROGRAM OUTCOME SUMMARY BY SERVICE 


NAVY 
Program Performance Cost Growth (3%) Schedule Slip (3%) 
1. S=-3A 1.04 130s L010 
2eTrident L 2Gi0 Ome 1.14 
3. Poseidon C-3 L060 N/A Or 
*. SSN 688 Pao 120 BES: 
5. Berdent C=$4 010 1206 ies 
6. BD 963 oo 1265 1.24 
7. CGN 38 1200 LG: 1.24 
8. Harpoon L200 Lako Less 
9. LAA 1 00 Takes sR, ety 
10. CVA®M 68 1.00 126 i. 3 
ii. GA 68 1200 Le25 1.24 
12. Condor 0.94 1.46 a, 
13. CleSer Oe)! 1.108 Pao2 
14. MK 48-1 0.98 Daeg 1.14 
15. Pe3c O2os 1.17 iPS 
6. Capter 1300 P56 i 
i7. Gpavrow III Im, 1.64 2.60 
18. PHM 00 1.84 1.41 
19. Phoenix 12e0 L428 1.06 
20. FFG 7 HG 8110) P45 ales 
21. A-7E Ore Se zi 122010 
22. B-2C Ors te, 1ie3i6 103 
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Program 
23. Phalanx 
2@. Fe15A 
25. Sidesinder 
(AIM 9-L) 
MEAN 
ARMY 
Program 
1. XM 198 
2. Strager 
8S. Hawk 
4. UTTAS 
5. Roeland 
6. XM 1 
1. Sa ERMSE - 
8. AAH 
2. MECV 
10. Dragon 
11. Copperhead 
12. TOW 
Ls. Lanmee 
le4e TAC IRE 
MEAN 


Performance 
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Performance 
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1.04 
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1.00 
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1.00 
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Cost Growth ($%) 
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Program Performance Cost Growth (3%) Schedule Slip($%) 
i) 8-16 a0 i010 1.00 
2. Bel Be be Dae lis. i> 
3% B- 10 L203 lew22 tek. 3 
oe 3A 1 02 eG ike Ibe 
5. -1L1A 1.00 0.94 1.00 
6. Maverick 1.00 iFe0 2 da 2 
Te Be 4 Lowi 1.40 iL teak 
8. F-14 170 lee 5 iOS 
9. Minuteman III Oi.282 dS BE One 
MEAN 1.006 1. va M60 
OVERALL MEAN Ose 1 2155 i 20 
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